Skip to content

fix(orchestrator): create /dev/fuse for dockerd; runner parity refinements - #164

Merged
Bnjoroge1 merged 1 commit into
mainfrom
fix/docker-fuse-device-node-and-runner-parity
Aug 20, 2026
Merged

fix(orchestrator): create /dev/fuse for dockerd; runner parity refinements#164
Bnjoroge1 merged 1 commit into
mainfrom
fix/docker-fuse-device-node-and-runner-parity

Conversation

@Bnjoroge1

@Bnjoroge1 Bnjoroge1 commented Aug 20, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Follow-up to #151 (the conformance campaign fixes).

  • docker: the krunfw guest kernel has fuse built in, but /dev boots as a plain tmpfs with no device nodes, so fuse-overlayfs (dockerd's fallback when its overlay probe fails) dies with fuse: device not found. The hook now creates /dev/fuse when the kernel lists fuse, letting dockerd auto-pick fuse-overlayfs (CoW) — on this kernel dockerd's overlay2 probe mount gets EINVAL and overlay2 is never viable, so the earlier approach fell back to vfs. vfs is now forced only when overlay fails AND fuse is absent. Combined with fix(orchestrator): conformance-campaign server and runtime fixes #151's start_dockerd() reap + exit 1 retry.
  • Verified live on the golden VM: hook creates /dev/fuse, dockerd 28.0.4 reports Storage Driver: fuse-overlayfs, and hello-world runs. Orchestrator tests pass.

Summary by cubic

Enable CoW storage for dockerd on krunfw guests by creating /dev/fuse and refining driver selection. Previously the overlay probe failed and we forced vfs; now we create /dev/fuse when the kernel supports fuse, let Docker auto-select fuse-overlayfs, and only force vfs when overlay fails and fuse is absent.

  • Create /dev/fuse when /proc/filesystems lists fuse; this lets dockerd fall back to fuse-overlayfs.
  • Do not force "overlay2" when the probe succeeds; leave the driver unset and only write /etc/docker/daemon.json when overriding. When the probe fails, set vfs only if /dev/fuse is missing.
  • Runner parity: set RLIMIT_NOFILE=524288, use the runner user’s ~/.cargo/bin in PATH, and add libsystemd0:amd64 for x86_64 Valkey binaries.

Written for commit 24a7426. Summary will update on new commits.

Review in cubic

Note

Create /dev/fuse in docker_start_command and let dockerd auto-select storage driver

  • After modprobe fuse, the startup script now creates /dev/fuse via mknod when the kernel lists fuse in /proc/filesystems and the device is missing
  • On a successful overlay probe, DRIVER is cleared instead of forcing overlay2; on a failed probe, vfs is only forced when /dev/fuse is absent, otherwise DRIVER is left unset so dockerd can auto-select fuse-overlayfs
  • /etc/docker/daemon.json is now written only when a storage driver is explicitly chosen, instead of always
  • Risk: environments that relied on the always-present daemon.json with a forced driver may now see dockerd auto-select a different storage driver when /dev/fuse exists and overlay fails

Macroscope summarized 24a7426.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Docker startup compatibility across environments with different kernel and storage capabilities.
    • Enabled FUSE support when available and preserved Docker’s automatic storage-driver selection.
    • Falls back to the vfs storage driver only when other supported options are unavailable.

@Bnjoroge1
fix(orchestrator): create /dev/fuse for dockerd; runner parity refine…
24a7426 
…ments

Follow-ups to the conformance campaign fixes (PR #151):

- docker: the krunfw guest kernel has fuse built in, but /dev boots as a
  plain tmpfs with no device nodes, so fuse-overlayfs (dockerd's fallback
  when its overlay probe fails) dies with 'fuse: device not found'. The
  hook now creates /dev/fuse when the kernel lists fuse, letting dockerd
  auto-pick fuse-overlayfs (CoW) — on this kernel dockerd's overlay2 probe
  mount gets EINVAL and overlay2 is never viable, so the earlier fix was
  falling back to vfs. vfs is now forced only when overlay fails AND fuse
  is absent.
- RLIMIT_NOFILE: 524288 instead of 1048576 — systemd's built-in hard
  default, which is what GitHub's runner service inherits (exact parity).
- guest PATH: cargo bin dir matches the runner user (/home/<user>/.cargo)
  instead of hardcoded /root/.cargo/bin, which the unprivileged runner
  cannot stat (nodejs/ci EACCES).
- multiarch shim: add libsystemd0:amd64 (valkey's x86_64 tarballs link
  libsystemd.so.0).

Verified live on the golden VM: hook creates /dev/fuse, dockerd 28.0.4
reports Storage Driver: fuse-overlayfs, and hello-world runs. 61
orchestrator tests pass.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 20, 2026

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Bnjoroge1
Bnjoroge1 merged commit 673bdfa into main Aug 20, 2026
6 of 8 checks passed
@Bnjoroge1
Bnjoroge1 deleted the fix/docker-fuse-device-node-and-runner-parity branch August 20, 2026 03:18
@coderabbitai

coderabbitai Bot commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: da06b45d-4776-4f70-a2bd-cc0067c9e752

📥 Commits

Reviewing files that changed from the base of the PR and between 0d92720 and 24a7426.

📒 Files selected for processing (1)
  • crates/preloop-orchestrator/src/lib.rs
📝 Walkthrough

Walkthrough

Docker startup now enables FUSE when supported, preserves Docker’s automatic storage-driver selection when overlay or FUSE is usable, and falls back to vfs only when neither option is available.

Changes

Docker storage driver selection

Layer / File(s) Summary
Docker startup driver selection
crates/preloop-orchestrator/src/lib.rs		 Startup provisions /dev/fuse when supported. It leaves the storage driver unset after a successful overlay probe. It writes an explicit vfs configuration only when overlay and FUSE are unavailable.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • preloopdev/preloop#151: Both changes modify Docker storage-driver selection in crates/preloop-orchestrator/src/lib.rs; this PR refines that behavior.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/docker-fuse-device-node-and-runner-parity

Comment @coderabbitai help to get the list of available commands.

Bnjoroge1 added a commit that referenced this pull request Aug 20, 2026
@Bnjoroge1
feat(update): resolve same-version installs by build commit, not bytes
0988039 
The equal-version branch byte-compared the installed binary against the
release asset and reinstalled on any drift. That clobbered a source build
from newer main: main reports the same version string as the latest tag
(no bump between tag and HEAD), so a build carrying #149/#151/#164 was
treated as drift and replaced with the stale release binary every hour.

Embed the build commit (build.rs reads git rev-parse HEAD) and expose it
in 'preloop version'. The updater now compares commits via the GitHub
compare API when versions are equal:

- release commit is ahead of installed (installed is an ancestor — the
  v0.30.2 deaf-runner case) -> reinstall
- installed is at or beyond the release -> keep
- diverged history (release cut from a dist commit off main, or a local
  build) or unverifiable (no embedded commit) -> keep; never clobber a
  real build on an ambiguous comparison

Keeps the version-greater upgrade path and the version-less stop
unchanged. Drops the byte-compare and its tests; adds decision-mapping
tests for ahead/behind/identical/diverged/unknown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Bnjoroge1